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ABSTRACT 


This thesis defines a campaign of experimentation to guide UTACC development 
from concept to reality. It also applies design methodologies to reduce costs and increase 
the quality, effectiveness, and speed of UTACC’s development. UTACC is a system of 
systems that teams Marines with unmanned robotic systems to reduce the Marine’s 
cognitive load and enhance mission accomplishment. Bringing UTACC from concept to 
reality requires extensive experimentation, but prior to this thesis no experimentation plan 
has existed. 

A series of UTACC theses have been written starting with a CONOPS. Then 
theses red-celled the CONOPS, explored Coactive Design methodology, analyzed UAV 
alternatives, and generated measures of effectiveness and performance for the system. 
Using information learned from the previous theses, the campaign of experimentation 
described in this thesis identifies key developmental relationships, associates measures 
with them, and organizes them in an incremental order. This thesis also emphasizes 
Coactive Design and Model Driven Software Development to reduce cost and improve 
the quality and flexibility of the system. The goal of the campaign is to provide a plan to 
develop a robust, cost-efficient system that Marines can use as a part of their team to 
increase victory on the battlefield. 
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EXECUTIVE SUMMARY 


Rapidly evolving technology has created information overload for decision 
makers. They are expected to pull specific and relevant information from a vast sea of 
data and then make decisions that impact Marines on the battlefield. The abundance of 
information can overwhelm warfighters and lead to degraded mission performance (Rice, 
Keim, & Chhabra, 2015, p. 3). The Unmanned Tactical Autonomous Control and 
Collaboration (UTACC) system is designed to enhance mission accomplishment while 
reducing the information load on the warfighter. UTACC consists of Marines, an 
unmanned ground component, and an unmanned aerial component acting as a team to 
accomplish future operations. 

This thesis is the sixth in a series of theses that support the Marine Corps 
Warfighting Laboratory’s (MCWL) development of the UTACC system. It describes 
how a campaign of experimentation will help system developers advance UTACC from 
concept to functional system. This thesis also uses design methodologies to facilitate 
efficient and effective system development. The campaign of experimentation satisfies 
the system engineering criteria of detail design and development. 

The campaign of experimentation is based on the concept of operations 
(CONOPS) that focuses system development. The campaign follows the Command and 
Control Research Program’s guiding principles of variety and replication. Their use 
ensures that a successful system is developed in a comprehensive and incremental way. 
Seven Critical Operational Issues (COI) provided in the planning worksheet attached in 
the appendixes are the foundation for the hypotheses driving experimentation. 

The campaign of experimentation uses Limited Technical Assessments (LTA) to 
organize experimentation into stages. Two LTAs are scheduled to take place each year. 
This allows a six month period for developers to conduct their own experimentation and 
address any issues that arise. MOEs and MOPs provide quantifiable standards to evaluate 
newly developed system capabilities. Using them as entrance and exit criteria ensures 



that replication occurs throughout the experimentation process and that the needs of the 
Marine Corps are met. 

Coactive design is one of the proposed design methodologies for UTACC 
developers. It focuses on human-machine interaction. The Marine Corps planning process 
is a framework for the UTACC coactive design process. Coactive design is used to 
identify the different variables associated with interdependence, tasks to be completed, 
and the relationship between the two. It is a unique combination of waterfall and spiral 
development models. The waterfall attributes of the process make it easier to follow and 
execute while the spiral model attributes facilitate adaptation throughout the design 
process (Satzinger, Jackson, & Burd, 2012, pp. 228, 230). The simplicity and flexibility 
of the coactive design method is a tremendous advantage for the UTACC development 
team. They can quickly identify interdependence variables and either add, subtract, or 
modify variables throughout the process. 

Model driven software development (MDSD) is the second design methodology 
recommended in this thesis. MDSD fits the DOD standard for software development 
outlined in DOD publication 5000.02 Operation of the Defense Acquisition System. The 
goals of MDSD are to increase development speed, improve the quality of the software 
created, improve software maintenance, increase reusability, manage the complexity of 
system, and increase interoperability (Stahl et al., 2006, p. 13-14). To accomplish these 
goals, software developers utilize models. The collection of models makes up an 
architecture. The architecture defines the system or system of systems and serves as a 
blueprint that software development teams can use as a foundation to create an 
application (Stahl et al., 2006, p. 22). The more detailed the architecture, the more 
thorough a blueprint is created, and the more efficient the software developers are 
because they can copy source coding rather than create new coding for a function. 

UTACC is a unique, innovative system that teams a Marine with a machine to 
reduce the Marines’ cognitive load and enhance mission success. Going from concept to 
reality requires robust, incremental experimentation. This thesis proposes a campaign of 
experimentation to accomplish exactly that. It focuses resources and the efforts of 
developers to create a system that serves as teammate on the battlefield instead of a tool. 
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I. 


INTRODUCTION 


A. UTACC VISION 

There are a large number of sensors and technologies used for war that are 
designed to increase mission accomplishment for the warfighter. These technologies have 
created information overload for the decision maker. Decision makers must pull specific 
information from a vast pool before making decisions that impact lives on the battlefield. 
This abundance of information can easily overwhelm the warfighter and lead to 
unintentional degraded mission performance (Rice, Keim, & Chhabra, 2015, p. 3). 

As stated by Rice et al. (2015), the Unmanned Tactical Autonomous Control and 
Collaboration (UTACC) system’s purpose is to enhance mission accomplishment while 
reducing the warfighter’s information overload. Collaborative autonomy accomplishes 
that goal. The system will consist of a team member, an unmanned ground component, 
and an unmanned aerial component acting as a team to accomplish future operations. 
System developers analyze operational context, possible missions with associated tasks, 
human system integration (HSI) factors, and data exchange requirements to understand 
the inherent complexities of the system and plan a way forward. The Marine Corps will 
need to innovate new technology to create a UTACC system that can function as an 
integral part of the team. 

This thesis is the sixth in a series of theses that discuss the Marine Corps 
Warfighting Laboratory’s (MCWL) development of the UTACC system. This thesis is 
similar to the previous five theses because UTACC development utilizes the incremental 
design process. MCWL’s mission is to rigorously explore and assess Marine Corps 
service concepts using war gaming, concept-based experimentation, technology 
assessments, and analysis. In addition to assessing service concepts, they aim to inform 
force development by validating, modifying, or rejecting concepts’ viability while 
identifying capability gaps and opportunities (“MCWL,” n.d., mission). The first thesis in 
this series developed a concept of operations for UTACC. The second thesis conducted a 
“Red Team” critique of the concept of operations (CONOPS). The third thesis explored 
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Marine and machine interdependence. The fourth thesis used CONOPS to develop 
specific measures of effectiveness (MOE) and measures of performance (MOP) for 
UTACC. The fifth thesis analyzes MCWL alternatives for an appropriate UTACC 
unmanned air vehicle. This thesis adapts the fourth thesis’s development of UTACC 
MOEs and MOPs to propose a coherent campaign of experimentation to ensure that 
UTACC focuses on the most important operational and technical concepts. 

B. RESEARCH QUESTION 

This thesis proposes a campaign of experimentation that system developers can 
use to advance UTACC from concept to functional system. This thesis will answer 
the following questions: 1) What is a campaign of experimentation and how should 
it apply to UTACC? 2) What are the key operational and technical elements of UTACC? 

3) How can one use MOEs and MOPs to determine if those elements have been met? 

4) How should the campaign of experimentation arrange the order of these elements? 

5) What are the associated entrance and exit criteria required to move from one element 
focus area to the next? 6) Can some elements be worked in parallel? 7) If so, which ones? 
These questions form the bedrock for understanding what a campaign of experimentation 
is, how UTACC developers can utilize it, and how the process of system development 
can occur. Successful UTACC system development requires the use of funds, new 
technologies, and a variety of novel innovations from diverse organizations. An 
incremental guide for experimentation will allow the Marine Corps to effectively focus 
the attention, skills, and resources needed to create UTACC (Alberts & Hayes, 2005, 
p. 63). The campaign of experimentation proposed in this thesis will attempt to 
accomplish just that. 

C. THESIS ORGANIZATION 

The thesis is organized into four chapters. The introduction explains how this 
thesis falls within the UTACC thesis series and uses content from the previous theses. 
The introduction explains the UTACC vision of creating a system that teams unmanned 
ground and aerial vehicles with Marines to reduce their cognitive load in combat. The 
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introduction introduces the research questions that guided thesis research and describes 
the intended impact of the thesis on UTACC system development. 

Chapter II, the literature review, summarizes background information regarding 
UTACC concepts and explains the intended impact of the system on the battlefield. The 
literature review explores autonomy in depth, a critical aspect of UTACC. It defines 
autonomy, the levels of autonomy, and the Department of Defense’s stance regarding 
autonomy and the potential benefits autonomy will offer in future conflicts. The chapter 
then explains measures of effectiveness and performance. Measures of effectiveness and 
performance are standards for the operation of a system. They focus experimentation 
efforts by functioning as entrance and exit criteria for those experiments. The chapter 
then describes of a campaign of experimentation and its various parts in detail. The 
campaign of experimentation is a system development roadmap designed to be a 
proactive, incremental guide to focus attention and resources (Alberts & Hayes, 2005, 
p. 63). The last two sub-sections of the literature review introduce different design 
methodologies developers can use to increase the speed, quality, and flexibility of the 
system throughout the development process. 

Chapter III explains key elements and design methodologies of the UTACC 
campaign of experimentation. The chapter explains and expands upon concepts 
introduced during the latter half of the literature review. The third chapter pulls heavily 
from the fourth and fifth theses in the series. The chapter highlights the importance of 
measures of effectiveness and performance, interdependence, and the coactive design 
process because they ensure a quality product is delivered to the warfighter. 

The fourth chapter of the current research moves away from the conceptual 

aspects of a campaign of experimentation and focuses on the practical application of the 

campaign. The chapter explains the logistical components of a recommended campaign 

of experimentation. The thesis includes a planning template for limited technical 

assessments and recommends a plan for future assessments. The chapter also lists 

entrance and exit criteria for each stage of the campaign. The chapter assesses which 

development efforts can be run in parallel to save system developers time. Finally, the 

chapter summarizes the thesis and recommends future areas of research for UTACC. The 
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fourth chapter will briefly describe potential research areas to orient developers toward 
future development. An appendix containing supporting documentation is also included. 


D. SECTION SUMMARY 

UTACC is a complex array of mature and developing technologies that will need 
to work together and in tandem with humans to accomplish a mission. Significant 
advancement in autonomy and additional technology is required before unmanned 
systems are capable of functioning as Marine teammates. This thesis explains how 
various concepts and methodologies can enhance the development of UTACC. The thesis 
is a launching point for development teams and is intended to generate the conversation 
and debate needed to move UTACC from a concept to a team member. 
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II. LITERATURE REVIEW 


A. BACKGROUND 

Unmanned Tactical Autonomous Collaboration and Control (UTACC) was 
developed to reduce Marines’ cognitive load. UTACC developers will increase the 
autonomy of unmanned ground and aerial systems to change the relationship between 
human and machine. Currently, unmanned system operators use a remote control to 
provide direct input to the system, called “human in the loop” operation (Rice, Keim, & 
Chhabra, 2015, p. 12). UTACC’s increased autonomy places the operator in a 
supervisory role over the system, a status called “human on the loop” (Rice et al., 2015, 
p. 12). Marines give the machine mission parameters, intent, and tasks and afterwards the 
machine executes those tasks autonomously. When a task is complete or a critical 
decision point is reached, the system notifies the human operator to receive 
acknowledgement or input before continuing with its mission. Putting the human “on the 
loop” allows the warfighter to focus on warfighting tasks. The push and pull of 
information between human and machine facilitates a teamwork relationship between 
man and machine. The UTACC program is developing software for unmanned 
systems that develops collaboration between systems and Marines (Gelhaus, 2015, p. 7). 
To do this, system developers will utilize “agent-based reasoning and semantic 
technologies to plan an optimized method to complete a task(s), goal(s), or performance 
measure(s); then set off to accomplish those with (or without) human partners (vice 
operators)” (Gelhaus, 2015, p. 7). 

To date, there have been two limited technical assessments to demonstrate and 
test the capabilities of the UTACC system. The first limited technical assessment (LTA) 
occurred on 26 February 2015 and the second on 18-22 April 2016; representatives from 
the Center for Naval Analysis (CNA) observed both and wrote reports (Gelhaus, 2015, 
p. 4). The first LTA was a proof of concept (POC) and occurred at Carnegie Mellon 
University’s (CMU) campus in Pittsburgh, Pennsylvania. The POC occurred in four 
stages. The first demonstrated current Marine cognitive load and how much time is 
required to accomplish a mission while operating a legacy Dragon Runner unmanned 
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ground vehicle (Gelhaus, 2015, p. 14). The second stage of the POC tested an unmanned 
ground vehicle developed by CMU. The focus of this part of the POC was mobility, 3D 
mapping, resource utilization, sensor capabilities, system prompts, and machine 
diagnostics (Gelhaus, 2015, p. 15). The third stage tested autonomous collaboration 
between CMU’s unmanned ground and aerial vehicles. The vehicles worked together 
using a shared 3D map to find a green pad on a desk oriented so the ground vehicle could 
not “see” it with its onboard sensors (Gelhaus, 2015, p. 17). The final stage of the POC 
tested the autonomous control of the CMU system and its ability to adjust to changes 
in its environment. It demonstrated how much time an operator spends accomplishing 
tasks without the system as compared to how much time is spent when an operator 
uses the system. Additionally, the final stage tested how the system reacted to 
environmental changes (Gelhaus, 2015, p. 18). CNA concluded that the autonomous 
systems were faster at accomplishing their tasks than their human-operated counterpart. 
Although CNA recommended further experimentation, it ultimately concluded that 
unmanned autonomous systems reduced Marines’ cognitive load (Gelhaus, 2015, p. 25). 

LTA 2 took place from 18-22 April 2016 in Ellis Hall aboard Marine Corps Base 
Quantico. It tested the UTACC hardware and its collaborative capabilities in a controlled 
environment. The experiment took place in a mock village with adjacent forest, river, and 
river crossing point inside the auditorium at Ellis Hall. LTA planners prepared eight 
scenarios to test UTACC, but only two were executed because software and hardware 
issues left little time for the rest. The first scenario tested the system’s ability to 
autonomously build a 3D model of the village and its surrounding area using data from 
the UGV and UAV. During the second scenario, the system searched for a high-value 
target while using and improving the map developed in the first scenario. The system 
used facial recognition software to identify the target. When either the UGV or the UAV 
found the target, the system requested confirmation from the Marine on the loop. When 
Marines gave confirmation and authorized target engagement, the system requested fires 
from an offshore platform. In this case, the offshore platform was the Navy’s Stiletto test 
ship tied up on the Potomac River. 
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UTACC was successful in both of the scenarios despite hardware limitations. The 
issues arose from the UGV and UAV platforms themselves, not the onboard equipment. 
The system used data from the UGV and UAV to quickly and autonomously build an 
accurate map of the designated operating area, satisfying the first scenario’s requirement. 
For the second scenario, the system effectively built on the first scenario map, identified 
its target, and, after receiving authorization, requested fire and engaged the target. 
Despite the limited scope of the LTA, the system demonstrated an acceptable level of 
hardware and collaborative capability. The system autonomously built a map and 
identified its designated targets using onboard equipment and accomplished all of its 
tasks through collaboration between the ground and aerial vehicles. LTA 2 reinforced 
UTACC’s viability as a system and its potential to improve Marines’ combat capability. 

B. AUTONOMY 

Little research explores the concept of UTACC; however, there is literature that 
describes automation and its application to future combat. Shaker and Wise (1988) 
explain the history of automation and robotics going back to World War I. The term 
autonomous, as used in robotics, is interpreted in a number of ways by people in the 
industry. Autonomy can range from direct control of a system to unmanned systems 
executing tasks with no human intervention (Bruemmer, Ferlis, Huang, Novak, Schultz, 
& Smith., 2004). Bruemmer et al. (2004) and Glotzbach (2004) define automation and 
provide guidelines for measuring levels of autonomy. UTACC is intended to develop a 
semi-autonomous system that enhances decision-making on the battlefield. The National 
Institute of Standards and Technology defines semi-autonomous as a form of operation 
where humans and machines execute missions by leveraging various levels of Human- 
Robot Interaction (HRI). In conjunction with the previous literary sources, Siegwart, 
Nourbakhsh, and Scaramuzza (2011) introduce the fundamentals of robotic and mobility 
autonomy. 

The Role of Autonomy in Department of Defense (DOD) Systems is critical to 
UTACC development. The Role of Autonomy in DOD Systems defines the current and 
future role of autonomy within the DOD. The document, created by the Department of 
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Defense, identifies opportunities and challenges in the integration of autonomous 
vehicles in the military (DOD, 2012). The DOD recognizes that the strength of unmanned 
aerial vehicles (UAV) is their ability to provide persistent intelligence, surveillance, and 
reconnaissance (ISR) on the battlefield. Increased autonomy enhances these capabilities 
and the safety of the system. Increased autonomy also reduces human error during take¬ 
off, flight, and landing, increasing the safety of the system. The safety and ISR that 
increased UAV autonomy provides reduces Marines’ cognitive load. UGVs similarly 
reduce Marines’ cognitive load. The Role of Autonomy in DOD Systems describes what 
UGVs are, their benefits, and how increased autonomy can improve them. It explains that 
UGVs’ major benefit is the standoff distance they provide the warfighter. It also mentions 
the need for a UGV capable of operating and making decisions in accordance with the 
rules of engagement (ROE) and a commander’s intent. At its completion, UTACC will 
function as a teammate that operates in accordance with Marine Corps doctrine and ROE. 

Gustavsson and Hieb (2013) address the challenges associated with integrating 
autonomous systems in the military. They introduced the “Operations Intent and 
Effects Model.” Their model outlines the integration of future Command and Control 
(C2) systems in the DOD to help the military recognize the benefits of automation. 
Lin, Beckey, and Abney (2008) add to this discussion by listing future missions and task 
sets for robotic systems and describing possible ethical implications of robotic systems’ 
use in war. 

Another important aspect of UTACC is Human-Robot Interaction (HRI). Groom 
and Nass (2008, p. 496) suggest that the following question guides HRI model 
development: “which human inabilities can the robot perform, and what organizational 
structure best supports both human and robots?” They note that future robotic teammates 
will have high levels of autonomy and coordination skills to assist their human 
counterparts, but that system functionality may be limited to specific environments unless 
explicitly designed otherwise. Integrating an HRI framework into system development 
creates broad system application and increases the success of a system in unpredictable 
environments (Groom & Nass, 2008, p. 483). 
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UTACC’s level of autonomy is significant to system development. As 
sophistication increases, human operators will assume a supervisory role over a system 
instead of actively controlling it (Chen & Barnes, 2014, p. 30). Maintaining a supervisory 
role over the system will be challenging if there is not a proper system interaction or 
interface to support the human operator (Chen & Barnes, 2014, p.23). Incoming sensor 
data will have to be relayed to C2 workstations and displayed in a comprehensible 
manner to be useful to the decision maker (Shattuck & Lewis Miller, 2006, p. 8). Key 
leaders must be supported with the most pertinent information, so that their perceptions, 
understanding, predictions, and decisions meet the requirements needed to accomplish 
the system’s goals (Shattuck & Lewis Miller, 2006, p. 6). Trafton et al. (2006) argue that 
integrating a computational cognitive model in a robotic system will increase its 
intelligence and create superior decision-maker support capability. 

Robot and Marine must exchange information to accomplish the goals of the 
UTACC system. Gold (2009) outlines four areas of information exchange: robot to 
human, environment to robot, human to robot, robot to environment. UTACC success 
also requires that information flows across the four areas in addition to a fifth, robot to 
robot. The UTACC system interacts with the environment through UGV and UAV on¬ 
board sensors while simultaneously transmitting and receiving information from other 
robots and humans via communication links. 

UTACC must be included within the Marine Corps Task List (MCTL) to evaluate 
capability gaps and determine efficiencies (Rice, Keim, & Chhabra, 2015, p. 16). The 
concepts of collaborative autonomy and interoperability do not exist in Marine Corps 
doctrine. However, Marine Corps Doctrinal Publication (MCDP)-l Warfighting along 
with MCDP-2 Intelligence, MCDP-3 Expeditionary Operations, MCDP-4 Logistics, 
MCDP-5 Planning, and MCDP-6 Command and Control are the fundamentals of Marine 
Corps warfighting and should be the basis for UTACC concepts. 

C. MOE/MOP 

In March 2014, the Marine Corps published Expeditionary Force 21 outlining the 
future of Marine Corps warfighting. It is a blueprint for the Marine Corps capabilities and 
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capacity decision-making processes (USMC, 2014a, p. 7). Expeditionary Force 21 
describes a modem force that actively integrates innovation and emerging technologies to 
create an advantage over future opponents (USMC, 2014a, p. 7). UTACC accomplishes 
that vision because it is an innovative technological concept that gives decision makers 
an edge over their enemies. 

UTACC will provide commanders with an advantage by integrating all 
warfighting functions (intelligence, maneuver, fires, logistics, force protection, command 
and control). Initial development, however, will focus on the intelligence warfighting 
function. Future iterations of the system will address the other functions. Development 
will begin by analyzing the Marine Corps Tasks (MCTs) because they will be used to 
build measures of effectiveness (MOE) and measures of performance (MOP) (Rice et al., 
2015, p. 17). The Joint Chief of Staffs directorate J-7 defines and explains MOEs and 
MOPs. The J-7 “is responsible for the six functions of joint force development: Doctrine, 
Education, Concept Development & Experimentation, Training, Exercises and Lessons 
Learned” (Glossary, n.d.). MOEs are created to analyze the effects, both good and bad, of 
a system on operations (JCS J7, 2011, p. III-4). They prompt an organization to assess its 
development efforts and track their progress toward accomplishing the system’s ultimate 
goals (Rushing & Kirkpatrick, 2016). MOPs are incremental ties that link system tasks to 
the MCTL. System developers use MOPs to align their efforts to the needs of the service 
acquiring the system (Rushing et. al, 2016). For UTACC, the Marine Corps Warfighting 
Lab and Naval Postgraduate School are the developing organizations and the Marine 
Corps is the acquiring service. System developers must analyze the MCTs and the 
develop MOEs and MOPs to ensure incremental UTACC system development (Rice et 
al., 2015). Incremental development requires extensive experimentation that should be 
structured pursuant to a campaign of experimentation. 

D. CAMPAIGN OF EXPERIMENTATION 

Experimentation is a cornerstone of DOD’s strategy to transform the current force 
into a technologically advanced, net centric force. UTACC experimentation will create a 
system that reinforces network capabilities, improves information sharing, and increases 
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situational awareness and mission effectiveness (Alberts, Hayes, Kirzl, Leedom, & 
Maxwell, 2005, pp. 8-9). Semi-autonomous system experimentation enhances the 
abilities of decision makers and supports the physical, information, and cognitive 
domains of net centric warfare (NCW) (Alberts et al., 2005, p. 10). The physical domain 
of NCW refers to the land, air, sea, and space spheres where warfare is conducted and 
combat effectiveness is judged. The information domain is where information is created, 
shared, manipulated, and where most command and control is communicated. The 
cognitive domain is in the mind of the individual warfighter. The cognitive domain 
consists of the tangibles of the tactics, techniques, and procedures in addition 
to the intangibles of leadership, situational awareness and more (Alberts et al., 2005, 
pp. 11-12). The DOD’s command and control research programs publication Code 
of Best Practices: Experimentation discusses the simultaneous relationships that take 
place between the domains of NCW to ensure mission success. Through experimentation, 
UTACC developers will leverage the relationships between domains to enhance 
mission accomplishment. 

A comprehensive campaign of experimentation will guide incremental UTACC 
development. A framework of discovery, hypothesis, and demonstration experiments in 
the campaign of experimentation accomplishes incremental system development. 
Discovery experiments are experiments that introduce new concepts, technologies, or 
systems to an environment for analysis (Alberts et al., 2005, p. 19). They determine the 
military viability of a system, potential employment methods, and conditions for or the 
limits of the systems’ use. Discovery experiments can be conducted in a number of 
ways—ranging from simulators to actual field usage—to facilitate innovation while 
reducing cost. Discovery experimentation is a precursor to hypothesis experimentation 
because hypotheses are created during this phase (Alberts et al., 2005, p. 20). 

Hypothesis testing is the next phase of experimentation. Hypothesis testing 
determines the limiting factors of the system and tests the system as a whole (Alberts et 
al., 2005, p. 22). The primary goal of hypothesis experimentation is to gain knowledge 
about possible variables that affect the functionality of the system. Initial hypothesis are 
reformed through subsequent hypothesis testing. Because hypotheses are tested and 
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reformulated through experimentation, system developers must carefully select initial 
hypotheses to test to avoid unnecessary complexity and data obfuscation (Alberts et al., 
2005, p. 22). The hypothesis testing phase is broken into two parts, a preliminary and 
refined segment. The preliminary phase of experimentation addresses the original 
hypothesis and the results refine future hypotheses. The refined hypothesis is then 
tested to both ensure the viability of the refined hypothesis and to ensure that the 
system functions under a variety of conditions (Alberts et al., 2005, p. 22). The 
results identify the viability of a system for military use and refine the system for 
demonstration experimentation. 

Demonstration experiments show the acquiring institution that the system 
enhances combat effectiveness and mission accomplishment under varying conditions 
(Alberts et al., 2005, p. 23). In contrast to hypothesis testing, demonstration experiments 
are not designed to create knowledge. Instead, they present known information to 
individuals who are not familiar with the system or data that has been created during 
the previous experimental phases. System developers should demonstrate the 
system under conditions specific to its use so that the experiment effectively conveys 
the viability of the capabilities of the system (Alberts et al., 2005, p. 23). System 
developers will identify a system’s specific conditions during the extensive discovery and 
hypothesis phases of the campaign. In all of the experiments, data collection is critical 
(Alberts et al., 2006, p. 22). 

For a campaign plan to be successful, it must be focused and identify objectives. 
MOEs and MOPs give UTACC developers focus and objectives. A campaign of 
experimentation is a framework that identifies key variables and relationships while 
effectively reinforcing MOEs and MOPs (Alberts et al., 2006, p. 69). In addition, a 
campaign of experimentation creates a balance between variety and replication (Alberts 
et al., 2006, p. 64). Experimentation variety allows system developers to identify issues 
that require follow-on experimentation (Alberts et al., 2006, p. 65). The amount of variety 
in a campaign of experimentation influences how robust the conclusions from the 
analysis of data collected are. Replication is a critical principle in the execution of a 
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campaign of experimentation because it shows that the results of the experimentation are 
not unique to a particular set of conditions (Alberts et al., 2006, p. 66). 

The UTACC campaign of experimentation follows the principles of phase 
transition: stage acknowledgement, nonlinear progress, resource availability, steering 
group recognition, and a broad scope of experimentation (Alberts et al., 2006, pp. 124- 
125). It is necessary for the system development team to acknowledge the transition from 
one stage of the campaign to the next because stage recognition creates knowledge 
continuity during experimentation. Continuity provides clarity for researchers if 
experimentation does not follow a linear path. Developers need to be aware of the 
possibility of nonlinear development progression to avoid frustration and allow 
developers to adjust to unexpected data discoveries. Senior steering groups allocate 
resources. Their authority should be recognized throughout the experimentation process. 
Lastly, awareness of the need for a broad scope of experimentation reinforces the 
system’s versatility (Alberts et al., 2005). 

E. COACTIVE DESIGN 

Coactive design is “a fresh design perspective built on interdependence, a more 
comprehensive understanding of interdependence, a model for human-machine systems, 
a design method, and a new tool to assist with system design and analysis called the 
Interdependence Analysis (IA) Table” (Zach, 2016, p. 16). Captain Matt Zach’s thesis, 
“Unmanned Tactical Autonomous Control and Collaboration Coactive Design,” 
introduced coactive design to the UTACC program. Coactive design is critical for the 
UTACC program because it defines where machine automation can be useful, identifies 
key machine human interactions, and improves the likelihood that unmanned systems 
will function as team members. Coactive design also describes how the close relationship 
between man and machine can be accomplished through the execution of a set of shared 
goals. Using interdependence analysis tables, designers are able to better understand the 
human-machine relationship and can therefore gain valuable insight into the coordination 
needed to accomplish different goals (Johnson, 2014, p. 46). There are three specific 
realms where interdependence exists: observability, predictability, and directability. 
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Coactive design’s design flexibility and relationship understanding ensures that humans 
and machines operate together to accomplish goals while broadening the machines’ 
capability to accomplish its immediate task and overall goals. 

F. MODEL DRIVEN SOFTWARE DEVELOPMENT 

Model driven software design (MDSD) is another design methodology that 
supports system development and flexibility. MDSD aims to increase development speed, 
improve software quality, improve software maintenance, increase reusability, manage 
system complexity, and increase interoperability (Stahl, Volter, Bettin, Haase, Helsen, & 
Czarnecki, 2006, p. 13). This methodology uses software models. A model is “an abstract 
representation of a system’s structure, function or behavior” (Stahl et al., 2006, p. 18). 
Models are used to document the structure of software for complex development projects 
(Stahl et al., 2006, p. 18). The models are related to the system through mapping and 
should include information about the system, rules for the system, and the definitions for 
terminology used in the system (Siegel, 2014, p. 5). The modeling process itself creates 
an architecture. The architecture defines the system or system of systems. UTACC is a 
system of systems (Siegel, 2014, p. 6). The UTACC system uses several complex 
software suites working collaboratively to complete a specific task and/or series of tasks. 
The models creating the architecture will provide the software developers the “exact 
meaning of program code” for the finalized UTACC product improving the software 
quality and development speed (Stahl et al., 2006, p. 14). 

Creating an architecture through the modeling process falls within the purview of 
Architecture-Centric MDSD (AC-MDSD). AC-MDSD is an approach developers can use 
to effectively organize complicated software structures. AC-MDSD is structured to assist 
the developer in avoided coding errors by increasing the quality, efficiency, and 
reusability of software (Stahl et al., 2006, p. 21). AC-MDSD uses the architecture 
developed through modeling as a blueprint that software development teams can use as a 
foundation to create an application (Stahl et al., 2006, p. 22). The more detailed the 
architecture, the more robust a blueprint is created. The more robust the blueprint, the 
more efficient software developers are because they can copy source coding rather than 
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create new coding for a function (Stahl et al., 2006, p. 22). The increased efficiency, 
quality, and reusability of software generated by AC-MDSD method benefits both short¬ 
term and long-term UTACC development. Software developers working on future 
iterations of the system can use or modify previously generated architectures and coding 
to meet future requirements. UTACC system development that employs AC-MDSD 
techniques will create an enduring system that meets the warfighting needs of the Marine 
Corps. 


G. SECTION CONCLUSION 

UTACC is a unique program within the Marine Corps because it transitions 
unmanned systems from Marines’ tools to their teammates. To accomplish this, the 
Marine Corps is increasing human-robot interaction by using and advancing the 
autonomy of unmanned systems. The literature review is a source for information about 
UTACC and how autonomy plays a role ensuring the success of the system. The next 
chapters of this thesis provide greater detail regarding the campaign of experimentation, 
its parts, and different design methods. This thesis includes templates for how to plan and 
organize future experimentation so that UTACC can evolve from a concept into an 
operational system. 
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III. RESEARCH METHODOLOGY 


A. SYSTEMS ENGINEERING APPROACH 

This thesis utilizes the research methodology known as the systems engineering 
approach. All prior UTACC theses have described and applied this methodology. This 
methodology originates from Benjamin S. Blanchard’s Systems Engineering 
Management (4th edition) textbook, where Blanchard outlines the systems engineering 
approach. Blanchard defines a system as “a construct or collection of different elements 
that together produce results not obtainable by the elements alone.” Rice et al. explain 
in their thesis, “Unmanned Tactical Autonomous Control and Collaboration Concept 
of Operations,” that UTACC will ideally function as a system of systems, collaborating 
to enhance mission accomplishment (Rice et al., 2015, p. 20). The campaign of 
experimentation for UTACC meets the systems engineering step of detail design and 
development. This thesis uses the concept of operations (CONOP) to outline the 
campaign of experimentation. 

B. CAMPAIGN OF EXPERIMENTATION 

The fundamental anatomy of a campaign of experimentation consists of a 
centralized focus, a set of objectives to gauge the success of the campaign, and variety 
and replication in how experiments are staged and hypothesis are refined (Alberts et al., 
2006, p. 69). Each objective has a set of measures associated with it to help analyze the 
effects of specific capabilities being tested and tie them back to essential Marine Corps 
tasks (Alberts et al., 2006, p. 69). Variety and replication are the guiding principles used 
for planning experimentation for the campaign of experimentation. They ensure that a 
successful system is developed in a comprehensive and incremental way (Alberts et al., 
2006, p. 64). 

UTACC will ultimately be a system of systems that reduces the cognitive load felt 
by Marines in combat, thereby enhancing Marines’ ability to accomplish missions. This 
will be accomplished by integrating autonomous robots into Marine Corps units (Rice et 
al., 2015, p. 20). This is the ultimate goal of UTACC and serves as a centralized focus 
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over the course of the campaign of experimentation. The next step is to identify the 
objectives that must be accomplished to make UTACC goals a reality. The broadest 
concrete objective of UTACC is to develop a system prototype and evaluate its 
capabilities (Alberts et al., 2006, p. 70). To create the system prototype, system 
developers assess system efficacy by comparing experiment data against entrance and 
exit criteria, otherwise known as MOEs and MOPs. Preliminary MOEs and MOPs for 
UTACC are provided in the fourth thesis of the series, “UTACC Measures of 
Performance and Measures of Effectiveness” (2016) and they are reproduced here in 
Appendix C. 

The previous chapter states that the DOD’s strategy for transforming the military 
into a net centric, technologically advanced force relies on extensive experimentation. 
Experimentation for UTACC will occur during both limited technical assessments 
(LTAs) and limited objective experiments (LOEs). Although the current thesis describes 
how system developers can structure LTAs, developers may use limited objective 
experiments (LOE) to advance UTACC knowledge and development. 

To thoroughly understand the campaign of experimentation, one must look at its 
most fundamental element, the experiment itself. The Command and Control Research 
Program’s (CCRP) book Experimentation (2005) explains that there are three types 
of experimentation: discovery, hypothesis and demonstration experiments. Figure 1 
illustrates the process of experimentation and what could result from it. 
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Figure 1. Process of Experimentation. Source: Alberts et al. (2005, p. 26). 

Discovery experiments are experiments where system developers introduce new 
concepts, technologies, or systems to an environment where their impact can be recorded 
and analyzed (Alberts et al., 2005, p. 19). Military technologists have traditionally relied 
on discovery experiments to determine the utility of a technology before giving it to end 
users to create a concept of operations (Alberts et al., 2005, p. 20). The UTACC system is 
different because a preliminary concept of operations has been formulated and 
technology is currently being adapted to fit that role. Although a preliminary concept of 
operations was created, experimentation for UTACC fits within the parameters of 
discovery experimentation because the experiments use mature technologies to develop 
new applications for those technologies and potentially refine the use of those 
technologies. Ultimately, UTACC discovery experimentation will determine whether the 
system is militarily viable, how the system can be used, and what conditions extend or 
limit the systems’ use (Alberts et al., 2005, p. 20). 
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After discovery experimentation, hypothesis experimentation investigates 
different variables that impact the system. Hypothesis testing is done in two phases, a 
preliminary phase and a refinement phase. Additionally, it requires a number of 
experiments to fully test the hypothesis (Alberts et al., 2005, p. 22). The preliminary 
phase of the experimentation addresses the hypothesis selected, with the results informing 
the hypothesis refining process. The newly refined hypothesis is then tested under a 
variety of conditions to verify the system’s efficacy (Alberts et al., 2005, p. 22). UTACC 
hypotheses are based on critical operational issues (COI) identified to steer the conduct of 
experimentation. A critical operational issue is “key operational effectiveness or 
suitability issues that must be examined in operational test and evaluation to determine 
the system’s capability to perform its mission” (Glossary, n.d.). This thesis proposes 
seven COIs: 1) Will the system reduce the cognitive load of the team? 2) Will the system 
render enhanced 3-dimensional reconnaissance products? 3) Will the system increase the 
safety of the team? 4) Will the system enhance identification and engagement of targets? 
5) Does the system operate in accordance with Marine Corps doctrine? 6) To what extent 
does the digital plan provide context to the machines as well as the Marines? 7) Does the 
system demonstrate flexibility to changes in the environment/plan? 

The final form of experimentation identified by the CCRP is demonstration 
experimentation. Demonstration experimentation will show that UTACC enhances 
combat effectiveness and mission accomplishment under a variety of conditions 
described in Chapter IV (Alberts et al., 2005, p. 23). Conducting the experiments in 
settings that the system will be used in will properly showcase its capabilities (Alberts et 
al., 2005, p. 23). These conditions are identified based on the nature of the previous two 
experimental phases (Alberts et al., 2005, p. 23). UTACC experimentation is designed in 
increments. As each increment builds on the last, the conditions the system is tested in 
evolve in complexity. The conditions or scenario remain the same for each stage of 
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experimentation, only the technological capabilities of the system change/evolve. Figure 
2 displays the nature of a campaign of experimentation and demonstrates how it follows 
an incremental incline as it progresses. As experimentation continues, the complexity of 
the system increases, refining the conditions for experimentation and advancing the 
knowledge of system capabilities (Alberts et al., 2005, p. 49). 



Figure 2. Nature of a Campaign of Experimentation. 

Source: Alberts et al. (2005, p. 49). 

Following this type of trajectory, the proposed campaign of experimentation will 
drive the progress of the UTACC system from concept to reality. 

C. COACTIVE DESIGN 

Coactive design is a design methodology that focuses on human-machine 
interaction which will be useful to UTACC developers. It is “a fresh design perspective 
built on interdependence, a more comprehensive understanding of interdependence, a 
model for human-machine systems, a design method, and a new tool to assist with system 
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design and analysis called the Interdependence Analysis (IA) Table” (Zach, 2016, p. 16). 
Captain Matt Zach’s thesis, “Unmanned Tactical Autonomous Control and Collaboration 
Coactive Design,” applies the design process to UTACC using the Marine Corps 
planning process BAMCIS (Begin the planning, Arrange reconnaissance, Make 
reconnaissance, Complete the plan, Issue the order, and Supervise). Appendix A models 
how BAMCIS applies to UTACC. This section gives an overview of the coactive design 
process and explains how its use throughout the execution of the campaign of 
experimentation will greatly enhance UTACC development. 

Dr. Matt Johnson of the Florida Institute of Human and Machine Cognition 
(IHMC) believes that the coactive design process is superior to others because it focuses 
on the interdependent relationship between human and machine (Zach, 2016, p. 17). The 
coactive design method captures the concepts of coordination, cooperation, and 
collaboration and conveys them in a requirements based format. The method consists of 
three processes: 1) the identification process, 2) selection and implementation, and 3) the 
evaluation of change processes (Zach, 2016, p. 24). Each process is then broken down 
into a series of subordinate processes. The inputs and outputs required for those sub¬ 
processes are defined. Figure 3 provides a visual representation of the coactive design 
process. 

Zach (2016) modified the original coactive design task analysis worksheets 
into UTACC interdependence analysis (IA) tables. Appendix B is an example of an IA 
table. The modified tables address the overarching tenets identified within the CONOPS 
while also identifying and addressing shortfalls that were not conceived by Rice et al. 
(2015) (Zach, 2016, p. 3). These tables assist developers in identifying the different 
variables associated with interdependence, tasks to be completed, and the relationship 
between the two. 
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Figure 3. Coactive Design Method. Source: Johnson (2014). 


23 








Coactive design is a unique combination of the waterfall and spiral design 
models. The waterfall attributes of the process make it easier to follow and execute while 
the spiral model attributes facilitate adaptation throughout the design process (Satzinger, 
Jackson, & Burd, 2012, pp. 228, 230). The simplicity and flexibility of the merged 
design methods give the UTACC development team an advantage because they can 
quickly identify interdependence variables and either add, subtract, or modify variables 
throughout the process. The method also helps developers generate what Dr. Johnson 
(2014) states is a better understanding of the human-machine relationship because 
developers must identify variables that impact that relationship. Therefore, developers 
gain valuable insight into the coordination needed to accomplish different goals. With the 
insight provided by the coactive design process, developers will be able to create a 
system that can better function as an autonomous robotic team member rather than a tool 
for Marines to operate on the battlefield. 

D. MODEL DRIVEN SOFTWARE DEVELOPMENT 

The UTACC campaign of experimentation described in this thesis also includes 
model driven software development as a design methodology. DOD publication 5000.02 
Operation of the Defense Acquisition System outlines the requirements for software 
development for DOD programs. It states that the development of software should be 
executed in a comprehensive, incremental, and efficient way in order to reduce cost 
and schedule overruns (DOD, 2015, p. 10). Model driven software development best fits 
the DOD standard for software development. The goals of MDSD are to increase 
development speed, improve the quality of the software created, improve software 
maintenance, increase reusability, manage the complexity of system, and increase 
interoperability (Stahl et al., 2006, p.13-14). To accomplish these goals, software 
developers utilize models to individually represent the system’s characteristics, l ik e its 
function and structure (Stahl et al., 2006, p.18). These models can be completed in a 
number of program languages; however, the most commonly used is the unified 
modeling language (UML) 2.5. As a result, this thesis recommends using UML 2.5. 
System developers create models by mapping the system, which includes defining system 

rules and defining the terms used in the system (Siegel, 2014, p. 5). 
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The most important step of the MDSD process is meta-model creation. The meta¬ 
model provides a description of the models’ structure and defines the modeling language 
(Stahl et al., 2006, p. 85). Classes make up the models’ structure and are a category that 
describes an object or thing. Each class contains common attributes or specific 
descriptors (Satzinger, 2012, p. 96). For UTACC, the classes for the meta-model would 
be derived from BAMCIS with each planning phase being its own class (Satzinger, 2012, 
p. 101). For example, begin planning is its own class, with attributes of system 
initialization and mission parameters (Rice et al., 2015, p. 39). 

After system developers create a meta-model, they should create an UMF profile 
to define the structure of the model and model constraints (Stahl et al., 2006, p. 19). With 
the meta-model complete, the models themselves are created. Fike the meta-model, the 
models are made up of classes with their own attributes, but they are created using the 
structure and language defined by the meta-model. With UTACC, the meta-model 
describes the use of BAMCIS for the creation of the models. The individual models will 
address the planning phases. The resultant models make up an architecture that serves as 
an overarching definition of the systems, or for UTACC a system of systems (Siegel, 
2014, p. 6). 

Architecture-centric MDSD is integrated into the UTACC software development 
process. AC-MDSD is structured to assist the developer in avoided coding errors by 
increasing the quality, efficiency, and reusability of software (Stahl et al., 2006, p. 21). 
The ability to attain these goals falls within DOD 5000.02 requirements for software 
development. By taking the time up front to create a thorough architecture, efficiency is 
increased. The architecture facilitates the creation of source code developers can use as a 
blueprint. The blueprints of source coding that are created and can be used, reused, or 
modified are called generative software architectures. As different UTACC software 
teams begin to build code that satisfies different functions within UTACC software 
architecture, other teams are able to copy that source coding rather than create it anew. 
The use of generative software architecture increases the efficiency of the development 
process, the interoperability of UTACC software, and the ability to easily modify the 
system in more sophisticated phases of development (Stahl et al., 2006, p. 22). 
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Generative software architecture is a key component of AC-MDSD because it facilitates 
the modular development of an application. 

Before software development begins, developers must read and understand the 
concepts introduced in the CONOPs. The structure of the model as well as the behaviors 
intended for the system to exhibit are built within the BAMCIS planning process and are 
described in the CONOPs. The architecture and models describe the system, and generate 
coding that will be used throughout software development. The ability to generate coding 
in the design process that can later be reused and/or modified is a tremendous strength of 
MDSD and helps to further increase speed, quality, and efficiency, while reducing cost. 

E. SECTION CONCLUSION 

The systems engineering approach is the research methodology most applicable to 
UTACC and for this thesis. The campaign of experimentation falls within the detailed 
design and development stage of the systems engineering model. The campaign starts 
with discovery experimentation, evolves to hypothesis experimentation, and finally 
extends to demonstration experimentation. Discovery experiments refine the uses of 
technologies being adapted for UTACC. Demonstration experiments will display the 
system’s developing capabilities, interoperability, and the interdependence with its 
human counter-parts. System developers will conduct experiments during LTAs when all 
development parties are present. 

A campaign of experimentation can incorporate different methodologies to 
produce the best results. The methodology to facilitate the quality, speed, and efficiency 
of interdependence development, as required by DOD 5000.02, for UTACC is coactive 
design. Using interdependence analysis tables, developers can identify the different 
UTACC specific variables that will move the technology from tool to teammate. Another 
method that improves quality, speed, and efficiency of software development is model 
driven software development. Utilizing the UTACC CONOPs, software developers will 
be able to thoroughly outline and build models and architectures that are in line with 
Marine Corps doctrine. Being able to create blueprints with associated coding that can be 
used by a number of development teams improves the efficiency of the development 
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process, saves money, and enhances the quality of the software. Ultimately, the campaign 
of experimentation and recommended methodologies take UTACC from concept to 
functional, mission enhancing reality. 
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IV. UTACC CAMPAIGN OF EXPERIMENTATION 


UTACC is a unique combination of software and hardware that functions 
autonomously while collaborating with Marines. The ability of the system to reduce 
Marines’ cognitive load is critical to mission and system success. A campaign of 
experimentation ensures UTACC meets these goals. The campaign of experimentation 
incrementally balances software and hardware capabilities in accordance with the 
CCRP’s principles of variety and replication. The campaign is structured around limited 
technical assessments because they serve as the primary setting for UTACC 
experimentation. 

A. ORGANIZATION 

Appendix D is a Microsoft Project Gantt chart outlining the proposed timeline, 
iterations, and focus of future LTAs. Because funding for UTACC is currently 
guaranteed until 2019, the chart begins where LTA two finished and runs through 2019. 
The chart is organized by LTAs and displays the primary focuses, design methodology, 
acceptance testing, follow-on acceptance testing, and correction times for each LTA. Two 
LTAs are scheduled to take place each year. This allows a six month period for 
developers to conduct their own experimentation/testing and address any issues that arise. 
Figure 4 is an example timeline for LTAs four and five. 



RacognlUvM 


p LTA Preparation Begins © Status Update (2 wee* <><**) 

© Acceptance testing *5 business days 
© Follow-on testing * S business days 
A Capability Experimentation 

f Limited Technology Assessment (LTA) • 5 business days 


Figure 4. LTA Timeline 
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Each LTA advances the ability of UTACC to function as a teammate by building 
on the accomplishments of the previous LTAs. LTA two took place indoors and 
demonstrated the ability of the system to autonomously build a 3D map, search for a 
designated target, and engage the target. The environment the experiment was conducted 
within was a simulated urban environment. The target was identified using facial 
recognition technology, and the target was engaged by an offshore platform. The 
unmanned air and ground vehicles both executed 3D mapping and facial recognition, that 
the air and ground vehicles also worked collaboratively to create a robust and accurate 
picture of both the physical and human terrain. The focus of LTA three is to transfer 
these capabilities from an indoor environment to an outdoor one. During LTA three, 
UTACC will be given the additional tasks of reacting to a new environmental variable 
and building a plan for the approach to a designated target or location. The system must 
accomplish the measures outlined in the MOEs and MOPs for this stage of 
experimentation before the system qualifies for the next stage. Coactive design and 
model driven software development are incorporated into LTA three and all follow-on 
LTAs as parallel efforts. 

LTA four experiments with voice command recognition capabilities to reduce 
Marine’s cognitive load when operating the system. Voice recognition allows the team 
leader to communicate with the system like he or she would with any member of their 
team, reducing and possibly eliminating the need for the Marine to interact with the 
system through a physical interface like a tablet. Voice recognition experimentation may 
take place in parallel with or after the measures accomplished in previous LTAs are 
repeated. It should be understood that the voice recognition referenced here does not 
mean full dialogue between the Marine and the machine. This stage of experimentation 
tests the system’s ability to receive basic, directional voice commands. The commands 
“forward,” “reverse,” “left,” “right,” and “stop” are the suggested goals. Anything more 
complex than this can be reached in future experimentation/development. 

LTA five develops Marine-machine communication by testing the system’s 
ability to auto-follow and understand hand and arm signals. Marines on patrol maintain 
formation and communicate non-verbally with one another. During early-stage 
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experimentation, the system should be able to recognize and maintain formation while 
following a designated member of the team. The system’s autonomous-follow ability 
allows Marine teammates to maintain situational awareness of their surroundings without 
distraction from the machine. Hand and arm signal recognition similarly increases 
Marines’ situational awareness, thereby enhancing combat capability. The UTACC 
CONOPS describe a scenario where a small reconnaissance team is inserted into a region. 
As the team moves through its area of operations, nonverbal communication maintains 
the covertness and safety of the team and mission. Hand and arm signals can be used to 
communicate everything from a moment’s pause to a change in patrol formation. At this 
stage, the hand and arm signals that LTA four tests are basic directional commands 
“forward,” “reverse,” “left,” “right,” and “stop.” This establishes a baseline that can be 
built on in future iterations of the system. Multiple means of communication and 
confidence in the functionality of the system facilitates Marines’ intuitive use of the 
machine. 

LTA six tests the planning and maintenance capabilities of the system, with a 
focus on threat analysis and self-diagnostics. Early stage planning ability testing begins in 
LTA three. In LTA three, the system must provide a basic plan for an approach to a 
designated target. The planning at that stage does not take into account potential threats 
to the team; it only recommends the most straightforward approach available. LTA six 
will test if the system can identify potential threats to the team and incorporate that 
information into a plan. The plan is presented to the team leader as a recommendation 
subject to acceptance, rejection, or modification. Because this is early stage 
experimentation, the expectation for threat analysis must be simple. At this stage, the 
system should be able to identify a basic linear danger area, like a road. The second focus 
of LTA six is self-diagnostic ability. Like any team member capable of communicating 
current physical condition information to the team leader, UTACC must be able to 
express its condition to the team leader. Information regarding the condition of the 
system is not only critical for maintenance reasons, but it also plays a role in the decision¬ 
making process of the team-leader. If the system has degraded for any reason, knowing 
this will allow the team leader to decide whether or not to use the system and its 


31 



remaining functional capabilities. Self-diagnostic capabilities means maintainers can 
quickly identify and repair issues, thereby saving time and money during maintenance 

Lastly, all LTAs should include acceptance and follow-on testing. During LTA 
two it became clear that experimentation and technology tests had not occurred prior to 
the LTAs execution. This resulted in slower progress. During acceptance and follow-on 
testing, project managers and the Marine Corps Warfighting Laboratory assess the 
progress of development teams and technology, facilitate coordination between team 
efforts, and identify capability gaps prior to the LTA. The current campaign of 
experimentation schedules acceptance testing 60 days prior to the LTA and follow-on 
testing 30 days prior to the LTA. Both acceptance and follow-on testing are scheduled to 
occur over a five-day period. Correction time occurs after acceptance testing and after 
follow-on testing. During correction time, development teams return to their respective 
design facilities and address any identified shortfalls before LTA execution. 

B. LIMITED TECHNICAL ASSESSMENTS 

The LTAs serve as the primary setting for all campaign experimentation. LTAs 
take place twice annually in periods of five business days in order to ensure that 
development teams have the time to acquire and build the technologies needed. The 
LTAs incorporate the principles of variety and replication. To ensure variety, 
experiments rotate which components of the system are being tested. To ensure 
replication, the experimentation environment remains constant. During later LTAs, 
measures met during prior LTAs must be repeated. The environment for LTAs three 
through six is a relatively flat, outdoor training area with a simulated urban setting. This 
environment will allow new variables to be introduced and reduces mobility challenges 
for the unmanned system at an early development stage. As the system matures and the 
capabilities of the unmanned systems increase, the environment must change to present 
new, realistic challenges to the system. Future environments should test the system’s 
functionality during increasingly difficult terrain and longer distances. 

Appendix E is a modified letter of instruction provided by 3D Low Altitude Air 
Defense Battalion. The letter is a template to organize and focus the requirements of 
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LTAs. The document serves as an easily understandable planning. The worksheet 
prompts developers to explain the situation and intent of the experimentation and LTA. It 
also prompts developers to create a mission statement to help focus the efforts of the 
development teams involved in the LTA. Developers also identify the critical operational 
issues being addressed through the template. The critical operational issues (COIs) serve 
as hypotheses for experimentation. The UTACC team has created seven COIs and they 
are disclosed in the modified template in the appendix. The template prompts developers 
to provide a concept of operations, clearly explaining how the LTA will be executed. 
Developers should use phases because they provide clear lines of delineation between the 
stages of experimentation. Lastly, the template requires that developers create 
coordinating instructions. These are instructions that highlight information important to 
all teams, such as entrance and exit criteria for the LTA and individual phases. Appendix 
E is filled out describing the recommended execution of LTA three. 

Developers must identify entrance criteria, exit criteria, and critical operational 
issues that are key prior to LTA execution. Measures of effectiveness and measures of 
performance are the entrance and exit criteria for UTACC experimentation. Appendix C 
displays MOEs and MOPs developed for UTACC. The MOEs and MOPs provide 
quantifiable standards to evaluate newly developed system capabilities. Using them as 
entrance and exit criteria ensures that replication occurs throughout the experimentation 
process and that the needs of the Marine Corps are met. 

In conjunction with entrance and exit criteria, COIs must be identified. The 
Defense Acquisition University defines critical operational issues as “key operational 
effectiveness or suitability issues that must be examined in operational test and evaluation 
to determine the system’s capability to perform its mission” (Glossary, n.d.). Seven COIs 
for UTACC are: 

1) Will the system reduce the cognitive load of the team? 

2) Will the system render enhanced 3-dimensional reconnaissance 
products? 

3) Will the system increase the safety of the team? 

4) Will the system enhance identification and engagement of targets? 
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5) Does the system operate in accordance with Marine Corps 
doctrine? 

6) To what extent does the digital plan provide context to the 
machines as well as the Marines? 

7) Does the system demonstrate flexibility to changes in the 
environment/plan? 

These COIs satisfy the requirement for all LTAs in the campaign of experimentation. 

C. SUMMARY 

UTACC is a unique system of systems that enhances mission accomplishment by 
reducing the Marines’ cognitive load in combat. It does this through increased autonomy 
of unmanned ground and air systems and improved interdependence between human and 
machine. Taking a machine from tool of war to teammate is not easily accomplished. A 
campaign of experimentation facilitates the accomplishment of this transition. The 
campaign of experimentation is based on the ideas introduced in the UTACC CONOPs 
and utilizes the MOEs and MOPs (Appendix C) developed in the fourth thesis of the 
UTACC series. The goal of the campaign is to create an incremental plan for 
experimentation that focuses development resources and the attention of developers to 
ensure the success of the system (Alberts et al., 2006, p. 63). The campaign of 
experimentation uses the guiding principles of variety and replication to ensure success. 
Variety allows developers to identify variables that may require further experimentation 
(Alberts et al., 2006, p. 65). Replication demonstrates that the results of experimentation 
are not unique to a specific set of conditions (Alberts et al., 2006, p. 65). Creating a 
balance between these principles facilitates a robust series of experimentation that 
advances system development (Alberts et al., 2006, p. 64). 

The campaign of experimentation takes advantage of limited technical 
assessments to provide the setting for experimentation. A Gantt chart (Appendix D) 
outlines the timeline, link, and subject of experimentation for each of the LTAs. With 
funding for the project guaranteed until 2019, the campaign of experimentation covers 
that time period. Two LTAs a year will provide developers the time needed to acquire 
technologies and conduct functionality testing on their own. The LTAs integrate coactive 
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design and model driven software development as preferred design methodologies. 
The LTA schedule also integrates acceptance and follow-on testing. These testing 
periods are important for the success of experimentation taking place during the LTAs. 
With acceptance and follow-on testing, MCWL can assess system development and 
technologies, identify capability gaps, and improve coordination between development 
teams prior to the LTA. 

The LTA schedule integrates variety and replication. Experimenting with 
different technologies during each LTA satisfies the requirement for variety. Experiments 
meet the requirement for replication when each LTA uses the same environment and each 
LTA repeats the accomplishments the prior LTA . The MOEs and MOPs provide the 
performance standards and entrance and exit criteria for each LTA. The environment for 
each LTA is a relatively flat, outdoor training area with a simulated urban setting. The 
same training area should be used for each LTA if possible. Appendix E provides a 
worksheet template to facilitate planning for the LTAs. The worksheet is a modified letter 
of instruction used by Marine Corps field units to communicate how the execution of an 
event, such as a field exercise, will occur. A unique requirement of the worksheet is that 
critical operational issues are identified. This thesis provides seven COIs that can remain 
consistent across all LTA efforts. The COIs provided address the breadth of the UTACC 
system as it currently exists. 

D. RECOMMENDATIONS 

This thesis plans for how future experimentation should occur. Going forward, the 
campaign of experimentation should be modified to incorporate future areas of 
experimentation so that future system development is incremental. UTACC’s success 
depends upon further experimentation on interdependence and the planning capability of 
the system. Interdependence enhances the system’s ability to serve as a team member. 
The smooth push and pull of information puts the Marine on the loop with the system. On 
the loop status reduces Marines’ cognitive load and increases mission accomplishment. 
Specifically, experiments testing different interfaces between the human and machine are 
essential to UTACC success. 
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Experiments focused on integrating information into a plan are critical for the 
system to be successful in the field. The system must be capable of taking on and 
simplifying tasks carried out by Marines to be successful. Planning requires time, energy, 
and resources and can take place during any stage of a mission. Spontaneous mission 
events require planning updates. UTACC’s ability to collect information on the terrain, to 
identify potential threats, and to incorporate threats and terrain into a plan for a team 
leader is invaluable. The system is merely suggesting a plan; the ultimate decision to 
accept, reject, or modify the plan rests with the Marine. Regardless of the team leader’s 
decision, having critical information consolidated and organized facilitates rapid planning 
and execution for Marines. With future experimentation, UTACC will evolve from a 
machine to a functioning Marine teammate. 
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APPENDIX A. BAMCIS MODEL 
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APPENDIX B. INTERDEPENDENCE ANALYSIS TABLE 


Option 1 | Option 2 | Option 3 


BAMCIS 

STEP 

Tasks 

Subtasks 

Capacities 

U 

A 

S 

u 

G 

S 

M 


U 

A 

S 

M 


U 

A 

S 

u 

G 

S 

OPD 

requirements 

Troop 

Leading 

Step 

(A) 

Main 

Task 

(A. 1) 

Subtask of 
Main 
Task (A) 

(A. 1.1) 
Capacity 
required for 
(A.1) 










Mechanisms, 
interface design 
elements, etc. that 
meet the 
Observability, 
Predictability, 
Directability 
requirements 
synthesized 
through the 
analysis of the 
interdependent 
teaming role 
alternatives. 

(A. 1.2) 
Capacity 
required for 
(A.1) 










(A.2) 

Subtask of 
Main 
Task (A) 

(A.2.1) 
Capacity 
required for 
(A.2) 










(A.3) 

Subtask of 
Main 
Task (A) 

(A.3.1) 
Capacity 
required for 
(A.3) 










(B) 

Main 

Task 

(B.l) 

Subtask of 
Main 
Task (B) 

(B. 1.1) 
Capacity 
required for 

(B.l) 










(B.2) 

Subtask of 
Main 
Task (B) 

(B.2.1) 
Capacity 
required for 
(B.2) 










(B.2.2) 

Capacity 
required for 

(B.2) 
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APPENDIX C. MOE AND MOP EXAMPLE 


Case 

Priority 

Objective 




MCT 

MCT 

Description 

MOP 

Result 

Unit 

Description 







1 

High 

Jointly 

Produce 

Map 




UTAC 

C 1.2 

Enter Mission 

Parameters 

Ml 

75 

% 

Input Orientation: Upload the 
present location, direction of attack 
and objective, and known key 
terrain data 

UTAC 

C 1.2 

Enter Mission 

Parameters 

M2 

80 

% 

Situation: Contains information on 
enemy (which will include SALUTE, 
DRAW-D, EMLCOA and EMDCOA) 
and friendly (which includes 
locations and missions of higher, 
adjacent and supporting units) 

UTAC 

C 1.2 

Enter Mission 

Parameters 

M3 

55 

% 

Mission: Upload the UXV's mission 
as related to the mission of the 
team (Who, What, When, Where, 
Why). Include tactical tasks. 

UTAC 

C 1.2 

Enter Mission 

Parameters 

M4 

60 

% 

Execution: Upload Concept of 
Operations (Commander's Intent, 
Scheme of Maneuver, Fire Support 
Plan), Tasks and Coordinating 
Instructions 

UTAC 

C 1.2 

Enter Mission 

Parameters 

M5 

67 

% 

Admin and Logistics: Define 
number and roles of humans and 
robots collaborating in team 
environment, and establish 
refueling and RTB points if different 
from origin 

2.2.1. 

2 

Conduct Area 

Reconnaissanc 

e 

Ml 

0.2 

Hrs 

From receipt of tasking, unit 
reconnaissance assets in place. 

2.2.1. 

2 

Conduct Area 

Reconnaissanc 

e 

M2 

Y 

Y/N 

Provide photographic and 
descriptive data of the Named Area 
of Interest to the Commander and 

staff. 

2.2.1. 

3 

Conduct Zone 

Reconnaissanc 

e 

Ml 

0.5 

Hrs 

From receipt of tasking, unit 
reconnaissance assets in place. 
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2.2.1. 

3 

Conduct Zone 

Reconnaissanc 

e 

M2 

N 

Y/N 

Provide photographic and 
descriptive data of the Named Area 
of Interest (NAI) to the Commander 
and staff. 

2.2.5. 

2 

Conduct 

Aviation 

Reconnaissanc 

e 

M3 

34 

% 

Of equipment ready and available 
to provide air reconnaissance 
operations. 

2.2.5. 

2 

Conduct 

Aviation 

Reconnaissanc 

e 

M4 

Y 

Y/N 

Product (sensor) 

dissemination/distribution network 
available. 

2.2.5. 

2 

Conduct 

Aviation 

Reconnaissanc 

e 

M7 

N 

Y/N 

Able to communicate relevant 
reconnaissance information using 
line-of-site (LOS)/beyond-line-of- 
site (BLOS) means. 

2.7 

Conduct 

Ground 

Reconnaissanc 

e and 

Surveillance 

M2 

45 

% 

Of equipment ready and available 
to provide reconnaissance and 
surveillance operations (i.e., 
communications, target 
designation, crew served weapons, 
infiltration/exfiltration equipment, 
mobility assets). 

2.7 

Conduct 

Ground 

Reconnaissanc 

e and 

Surveillance 

M4 

1 

Hrs 

From receipt of tasking, unit 
reconnaissance/surveillance assets 
in place. 

2.7 

Conduct 

Ground 

Reconnaissanc 

e and 

Surveillance 

M5 

70 

% 

Of collection requirements fulfilled 
by reconnaissance/surveillance 
assets. 

UTAC 

C 2.1 

Conduct Initial 
Mapping - 
Depart 

Friendly Lines 

Ml 

Y 

Y/N 

Resolve airspace deconfliction and 
meet safety threshold for launch. 

UTAC 

C 2.1 

Conduct Initial 
Mapping - Geo 
Scan 

M2 

2 

Hrs 

Understand the size of area to scan 
between origin and objective. Scan 
the area between origin and 
objective for specific geographic 
features. Scan objective area for 
basic geography. Execute mapping 
protocol. Generate actionable 
information. 

UTAC 

Conduct Initial 

M3 

1.5 

Hrs 

Transmit map info, identify urban 
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C 2.1 

Mapping - 
Build Map 




and wooded areas, identify masked 
areas, and fill in gaps in intel. 

UTAC 

C 2.2 

Select 

Emphasis Area 
- Review Map 

Ml 

0.5 

Hrs 

Different angle, higher resolution, 
different sensor, camera direction, 
multiple directions. Identify 
potential danger areas, routes, LZ's, 
water features...etc. 

2.2.1. 

1 

Conduct Route 

Reconnaissanc 

e 

M4 

1 

Hrs 

To conduct initial route study 
(dismounted/mounted). 

UTAC 

C 3.1 

Conduct 

Detailed 

Mapping 

Ml 

70 

% 

Scan Emphasis Areas. Execute 
detailed mapping protocol (the 
protocol will be different for why 
we selected the area for additional 
emphasis) i.e. If for LZ, execute the 
LZ protocol, if for route then etc. 
Build detailed map collaboratively. 

UTAC 

C 3.2 

MCOO 

M2 

25 

% 

Depict Surface Drainage. Depict 
water sources (width, depth, 
velocity, bank slope, height, and 
potential flood zones) 

2.2.1. 

1 

Conduct Route 

Reconnaissanc 

e 

M2 

Y 

Y/N 

Route/road confirmed. 

1.5 

High 

Jointly 

Produce 

Map of 
Alternate 

Environment 




UTAC 

C 1.2 

Enter Mission 

Parameters 

Ml 

75 

% 

Input Orientation: Upload the 
present location, direction of attack 
and objective, and known key 
terrain data 

UTAC 

C 1.2 

Enter Mission 

Parameters 

M2 

80 

% 

Situation: Contains information on 
enemy (which will include SALUTE, 
DRAW-D, EMLCOA and EMDCOA) 
and friendly (which includes 
locations and missions of higher, 
adjacent and supporting units) 

UTAC 

C 1.2 

Enter Mission 

Parameters 

M3 

55 

% 

Mission: Upload the UxV's mission 
as related to the mission of the 
team (Who, What, When, Where, 
Why). Include tactical tasks. 

UTAC 

C 1.2 

Enter Mission 

Parameters 

M4 

60 

% 

Execution: Upload Concept of 
Operations (Commander's Intent, 
Scheme of Maneuver, Fire Support 
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Plan), Tasks and Coordinating 
Instructions 

UTAC 

C 1.2 

Enter Mission 

Parameters 

M5 

67 

% 

Admin and Logistics: Define 
number and roles of humans and 
robots collaborating in team 
environment, and establish 
refueling and RTB points if different 
from origin 

2.2.1. 

2 

Conduct Area 

Reconnaissanc 

e 

Ml 

0.2 

Hrs 

From receipt of tasking, unit 
reconnaissance assets in place. 

2.2.1. 

2 

Conduct Area 

Reconnaissanc 

e 

M2 

Y 

Y/N 

Provide photographic and 
descriptive data of the Named Area 
of Interest to the Commander and 

staff. 

2.2.1. 

3 

Conduct Zone 

Reconnaissanc 

e 

Ml 

0.5 

Hrs 

From receipt of tasking, unit 
reconnaissance assets in place. 

2.2.1. 

3 

Conduct Zone 

Reconnaissanc 

e 

M2 

N 

Y/N 

Provide photographic and 
descriptive data of the Named Area 
of Interest (NAI) to the Commander 
and staff. 

2.2.5. 

2 

Conduct 

Aviation 

Reconnaissanc 

e 

M3 

34 

% 

Of equipment ready and available 
to provide air reconnaissance 
operations. 

2.2.5. 

2 

Conduct 

Aviation 

Reconnaissanc 

e 

M4 

Y 

Y/N 

Product (sensor) 

dissemination/distribution network 
available. 

2.2.5. 

2 

Conduct 

Aviation 

Reconnaissanc 

e 

M7 

N 

Y/N 

Able to communicate relevant 
reconnaissance information using 
line-of-site (LOS)/beyond-line-of- 
site (BLOS) means. 

2.7 

Conduct 

Ground 

Reconnaissanc 

e and 

Surveillance 

M2 

45 

% 

Of equipment ready and available 
to provide reconnaissance and 
surveillance operations (i.e., 
communications, target 
designation, crew served weapons, 
infiltration/exfiltration equipment, 
mobility assets). 

2.7 

Conduct 

Ground 

Reconnaissanc 

e and 

Surveillance 

M4 

1 

Hrs 

From receipt of tasking, unit 
reconnaissance/surveillance assets 
in place. 
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2.7 

Conduct 

Ground 

Reconnaissanc 

e and 

Surveillance 

M5 

70 

% 

Of collection requirements fulfilled 
by reconnaissance/surveillance 
assets. 

UTAC 

C 2.1 

Conduct Initial 
Mapping - 
Depart 

Friendly Lines 

Ml 

Y 

Y/N 

Resolve airspace de-confliction and 
meet safety threshold for launch. 

UTAC 

C 2.1 

Conduct Initial 
Mapping - Geo 
Scan 

M2 

2 

Hrs 

Understand the size of area to scan 
between origin and objective. Scan 
the area between origin and 
objective for specific geographic 
features. Scan objective area 
for basic geography. Execute 
mapping protocol. Generate 
actionable information. 

UTAC 

C 2.1 

Conduct Initial 
Mapping - 
Build Map 

M3 

1.5 

Hrs 

Transmit map info, identify urban 
and wooded areas, identify masked 
areas, and fill in gaps in intel. 

UTAC 

C 2.2 

Select 

Emphasis Area 
- Review Map 

Ml 

0.5 

Hrs 

Different angle, higher resolution, 
different sensor, camera direction, 
multiple directions. Identify 
potential danger areas, routes, LZ's, 
water features...etc. 

2.2.1. 

1 

Conduct Route 

Reconnaissanc 

e 

M4 

1 

Hrs 

To conduct initial route study 
(dismounted/mounted). 

UTAC 

C 3.1 

Conduct 

Detailed 

Mapping 

Ml 

70 

% 

Scan Emphasis Areas. Execute 
detailed mapping protocol (the 
protocol will be different for why 
we selected the area for additional 
emphasis) i.e. If for LZ, execute the 

LZ protocol, if for route then etc. 
Build detailed map collaboratively. 

UTAC 

C 3.2 

MCOO 

M2 

25 

% 

Depict Surface Drainage. Depict 
water sources (width, depth, 
velocity, bank slope, height, and 
potential flood zones) 

2.2.1. 

1 

Conduct Route 

Reconnaissanc 

e 

M2 

Y 

Y/N 

Route/road confirmed. 

2 

High 

Target Only 
Visible to 

UGV 




2.2.1. 

Conduct Area 

Ml 

0.2 

Hrs 

From receipt of tasking, unit 
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2 

Reconnaissanc 

e 




reconnaissance assets in place. 

2.2.1. 

2 

Conduct Area 

Reconnaissanc 

e 

M2 

Y 

Y/N 

Provide photographic and 
descriptive data of the Named Area 
of Interest to the Commander and 

staff. 

2.2.1. 

3 

Conduct Zone 

Reconnaissanc 

e 

Ml 

0.5 

Hrs 

From receipt of tasking, unit 
reconnaissance assets in place. 

2.2.1. 

3 

Conduct Zone 

Reconnaissanc 

e 

M2 

N 

Y/N 

Provide photographic and 
descriptive data of the Named Area 
of Interest (NAI) to the Commander 
and staff. 

2.2.5. 

2 

Conduct 

Aviation 

Reconnaissanc 

e 

M3 

34 

% 

Of equipment ready and available 
to provide air reconnaissance 
operations. 

2.2.5. 

2 

Conduct 

Aviation 

Reconnaissanc 

e 

M4 

Y 

Y/N 

Product (sensor) 

dissemination/distribution network 
available. 

2.2.5. 

2 

Conduct 

Aviation 

Reconnaissanc 

e 

M7 

N 

Y/N 

Able to communicate relevant 
reconnaissance information using 
line-of-site (LOS)/beyond-line-of- 
site (BLOS) means. 

2.7 

Conduct 

Ground 

Reconnaissanc 

e and 

Surveillance 

M2 

45 

% 

Of equipment ready and available 
to provide reconnaissance and 
surveillance operations (i.e., 
communications, target 
designation, crew served weapons, 
infiltration/exfiltration equipment, 
mobility assets). 

2.7 

Conduct 

Ground 

Reconnaissanc 

e and 

Surveillance 

M4 

1 

Hrs 

From receipt of tasking, unit 
reconnaissance/surveillance assets 
in place. 

2.7 

Conduct 

Ground 

Reconnaissanc 

e and 

Surveillance 

M5 

70 

% 

Of collection requirements fulfilled 
by reconnaissance/surveillance 
assets. 

2.2 

Collect Data 

and 

Intelligence 

Ml 

25 

% 

Of targets accurately identified. 

2.2 

Collect Data 

M2 

25 

% 

Of targets accurately located. 


46 









and 

Intelligence 





3 

High 

Target Only 
Visible to 

UAV 




2.2.1. 

2 

Conduct Area 

Reconnaissanc 

e 

Ml 

0.2 

Hrs 

From receipt of tasking, unit 
reconnaissance assets in place. 

2.2.1. 

2 

Conduct Area 

Reconnaissanc 

e 

M2 

Y 

Y/N 

Provide photographic and 
descriptive data of the Named Area 
of Interest to the Commander and 

staff. 

2.2.1. 

3 

Conduct Zone 

Reconnaissanc 

e 

Ml 

0.5 

Hrs 

From receipt of tasking, unit 
reconnaissance assets in place. 

2.2.1. 

3 

Conduct Zone 

Reconnaissanc 

e 

M2 

N 

Y/N 

Provide photographic and 
descriptive data of the Named Area 
of Interest (NAI) to the Commander 
and staff. 

2.2.5. 

2 

Conduct 

Aviation 

Reconnaissanc 

e 

M3 

34 

% 

Of equipment ready and available 
to provide air reconnaissance 
operations. 

2.2.5. 

2 

Conduct 

Aviation 

Reconnaissanc 

e 

M4 

Y 

Y/N 

Product (sensor) 

dissemination/distribution network 
available. 

2.2.5. 

2 

Conduct 

Aviation 

Reconnaissanc 

e 

M7 

N 

Y/N 

Able to communicate relevant 
reconnaissance information using 
line-of-site (LOS)/beyond-line-of- 
site (BLOS) means. 

2.7 

Conduct 

Ground 

Reconnaissanc 

e and 

Surveillance 

M2 

45 

% 

Of equipment ready and available 
to provide reconnaissance and 
surveillance operations (i.e., 
communications, target 
designation, crew served weapons, 
infiltration/exfiltration equipment, 
mobility assets). 

2.7 

Conduct 

Ground 

Reconnaissanc 

e and 

Surveillance 

M4 

1 

Hrs 

From receipt of tasking, unit 
reconnaissance/surveillance assets 
in place. 

2.7 

Conduct 

Ground 

Reconnaissanc 

M5 

70 

% 

Of collection requirements fulfilled 
by reconnaissance/surveillance 
assets. 
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e and 

Surveillance 





2.2 

Collect Data 

and 

Intelligence 

Ml 

25 

% 

Of targets accurately identified. 

2.2 

Collect Data 

and 

Intelligence 

M2 

25 

% 

Of targets accurately located. 

4 

High 

Target Not 
Present 




2.2.1. 

2 

Conduct Area 

Reconnaissanc 

e 

Ml 

0.2 

Hrs 

From receipt of tasking, unit 
reconnaissance assets in place. 

2.2.1. 

2 

Conduct Area 

Reconnaissanc 

e 

M2 

Y 

Y/N 

Provide photographic and 
descriptive data of the Named Area 
of Interest to the Commander and 

staff. 

2.2.1. 

3 

Conduct Zone 

Reconnaissanc 

e 

Ml 

0.5 

Hrs 

From receipt of tasking, unit 
reconnaissance assets in place. 

2.2.1. 

3 

Conduct Zone 

Reconnaissanc 

e 

M2 

N 

Y/N 

Provide photographic and 
descriptive data of the Named Area 
of Interest (NAI) to the Commander 
and staff. 

2.2.5. 

2 

Conduct 

Aviation 

Reconnaissanc 

e 

M3 

34 

% 

Of equipment ready and available 
to provide air reconnaissance 
operations. 

2.2.5. 

2 

Conduct 

Aviation 

Reconnaissanc 

e 

M4 

Y 

Y/N 

Product (sensor) 

dissemination/distribution network 
available. 

2.2.5. 

2 

Conduct 

Aviation 

Reconnaissanc 

e 

M7 

N 

Y/N 

Able to communicate relevant 
reconnaissance information using 
line-of-site (LOS)/beyond-line-of- 
site (BLOS) means. 

2.7 

Conduct 

Ground 

Reconnaissanc 

e and 

Surveillance 

M2 

45 

% 

Of equipment ready and available 
to provide reconnaissance and 
surveillance operations (i.e., 
communications, target 
designation, crew served weapons, 
infiltration/exfiltration equipment, 
mobility assets). 

2.7 

Conduct 

Ground 

Reconnaissanc 

M4 

1 

Hrs 

From receipt of tasking, unit 
reconnaissance/surveillance assets 
in place. 
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e and 

Surveillance 





2.7 

Conduct 

Ground 

Reconnaissanc 

e and 

Surveillance 

M5 

70 

% 

Of collection requirements fulfilled 
by reconnaissance/surveillance 
assets. 

2.2 

Collect Data 

and 

Intelligence 

Ml 

25 

% 

Of targets accurately identified. 

2.2 

Collect Data 

and 

Intelligence 

M2 

25 

% 

Of targets accurately located. 

4.5 

Low 

Evasive 

Target 




2.2.1. 

2 

Conduct Area 

Reconnaissanc 

e 

Ml 

0.2 

Hrs 

From receipt of tasking, unit 
reconnaissance assets in place. 

2.2.1. 

2 

Conduct Area 

Reconnaissanc 

e 

M2 

Y 

Y/N 

Provide photographic and 
descriptive data of the Named Area 
of Interest to the Commander and 

staff. 

2.2.1. 

3 

Conduct Zone 

Reconnaissanc 

e 

Ml 

0.5 

Hrs 

From receipt of tasking, unit 
reconnaissance assets in place. 

2.2.1. 

3 

Conduct Zone 

Reconnaissanc 

e 

M2 

N 

Y/N 

Provide photographic and 
descriptive data of the Named Area 
of Interest (NAI) to the Commander 
and staff. 

2.2.5. 

2 

Conduct 

Aviation 

Reconnaissanc 

e 

M3 

34 

% 

Of equipment ready and available 
to provide air reconnaissance 
operations. 

2.2.5. 

2 

Conduct 

Aviation 

Reconnaissanc 

e 

M4 

Y 

Y/N 

Product (sensor) 

dissemination/distribution network 
available. 

2.2.5. 

2 

Conduct 

Aviation 

Reconnaissanc 

e 

M7 

N 

Y/N 

Able to communicate relevant 
reconnaissance information using 
line-of-site (LOS)/beyond-line-of- 
site (BLOS) means. 

2.7 

Conduct 

Ground 

Reconnaissanc 

e and 

Surveillance 

M2 

45 

% 

Of equipment ready and available 
to provide reconnaissance and 
surveillance operations (i.e., 
communications, target 
designation, crew served weapons, 
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infiltration/exfiltration equipment, 
mobility assets). 

2.7 

Conduct 

Ground 

Reconnaissanc 

e and 

Surveillance 

M4 

1 

Hrs 

From receipt of tasking, unit 
reconnaissance/surveillance assets 
in place. 

2.7 

Conduct 

Ground 

Reconnaissanc 

e and 

Surveillance 

M5 

70 

% 

Of collection requirements fulfilled 
by reconnaissance/surveillance 
assets. 

2.2 

Collect Data 

and 

Intelligence 

Ml 

25 

% 

Of targets accurately identified. 

2.2 

Collect Data 

and 

Intelligence 

M2 

25 

% 

Of targets accurately located. 

5 

High 

Only 

Incorrect 

Target(s) 

Present 




2.2.1. 

2 

Conduct Area 

Reconnaissanc 

e 

Ml 

0.2 

Hrs 

From receipt of tasking, unit 
reconnaissance assets in place. 

2.2.1. 

2 

Conduct Area 

Reconnaissanc 

e 

M2 

Y 

Y/N 

Provide photographic and 
descriptive data of the Named Area 
of Interest to the Commander and 

staff. 

2.2.1. 

3 

Conduct Zone 

Reconnaissanc 

e 

Ml 

0.5 

Hrs 

From receipt of tasking, unit 
reconnaissance assets in place. 

2.2.1. 

3 

Conduct Zone 

Reconnaissanc 

e 

M2 

N 

Y/N 

Provide photographic and 
descriptive data of the Named Area 
of Interest (NAI) to the Commander 
and staff. 

2.2.5. 

2 

Conduct 

Aviation 

Reconnaissanc 

e 

M3 

34 

% 

Of equipment ready and available 
to provide air reconnaissance 
operations. 

2.2.5. 

2 

Conduct 

Aviation 

Reconnaissanc 

e 

M4 

Y 

Y/N 

Product (sensor) 

dissemination/distribution network 
available. 

2.2.5. 

2 

Conduct 

Aviation 

M7 

N 

Y/N 

Able to communicate relevant 
reconnaissance information using 
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Reconnaissanc 

e 




line-of-site (LOS)/beyond-line-of- 
site (BLOS) means. 

2.7 

Conduct 

Ground 

Reconnaissanc 

e and 

Surveillance 

M2 

45 

% 

Of equipment ready and available 
to provide reconnaissance and 
surveillance operations (i.e., 
communications, target 
designation, crew served weapons, 
infiltration/exfiltration equipment, 
mobility assets). 

2.7 

Conduct 

Ground 

Reconnaissanc 

e and 

Surveillance 

M4 

1 

Hrs 

From receipt of tasking, unit 
reconnaissance/surveillance assets 
in place. 

2.7 

Conduct 

Ground 

Reconnaissanc 

e and 

Surveillance 

M5 

70 

% 

Of collection requirements fulfilled 
by reconnaissance/surveillance 
assets. 

2.2 

Collect Data 

and 

Intelligence 

Ml 

25 

% 

Of targets accurately identified. 

2.2 

Collect Data 

and 

Intelligence 

M2 

25 

% 

Of targets accurately located. 

6 

High 

Both Correct 

and Incorrect 

Targets 

Present 




2.2.1. 

2 

Conduct Area 

Reconnaissanc 

e 

Ml 

0.2 

Hrs 

From receipt of tasking, unit 
reconnaissance assets in place. 

2.2.1. 

2 

Conduct Area 

Reconnaissanc 

e 

M2 

Y 

Y/N 

Provide photographic and 
descriptive data of the Named Area 
of Interest to the Commander and 

staff. 

2.2.1. 

3 

Conduct Zone 

Reconnaissanc 

e 

Ml 

0.5 

Hrs 

From receipt of tasking, unit 
reconnaissance assets in place. 

2.2.1. 

3 

Conduct Zone 

Reconnaissanc 

e 

M2 

N 

Y/N 

Provide photographic and 
descriptive data of the Named Area 
of Interest (NAI) to the Commander 
and staff. 

2.2.5. 

2 

Conduct 

Aviation 

Reconnaissanc 

M3 

34 

% 

Of equipment ready and available 
to provide air reconnaissance 
operations. 
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e 





2.2.5. 

2 

Conduct 

Aviation 

Reconnaissanc 

e 

M4 

Y 

Y/N 

Product (sensor) 

dissemination/distribution network 
available. 

2.2.5. 

2 

Conduct 

Aviation 

Reconnaissanc 

e 

M7 

N 

Y/N 

Able to communicate relevant 
reconnaissance information using 
line-of-site (LOS)/beyond-line-of- 
site (BLOS) means. 

2.7 

Conduct 

Ground 

Reconnaissanc 

e and 

Surveillance 

M2 

45 

% 

Of equipment ready and available 
to provide reconnaissance and 
surveillance operations (i.e., 
communications, target 
designation, crew served weapons, 
infiltration/exfiltration equipment, 
mobility assets). 

2.7 

Conduct 

Ground 

Reconnaissanc 

e and 

Surveillance 

M4 

1 

Hrs 

From receipt of tasking, unit 
reconnaissance/surveillance assets 
in place. 

2.7 

Conduct 

Ground 

Reconnaissanc 

e and 

Surveillance 

M5 

70 

% 

Of collection requirements fulfilled 
by reconnaissance/surveillance 
assets. 

2.2 

Collect Data 

and 

Intelligence 

Ml 

25 

% 

Of targets accurately identified. 

2.2 

Collect Data 

and 

Intelligence 

M2 

25 

% 

Of targets accurately located. 

8 

High 

Start Hunt 
for Target at 
Suspected 
Location 




2.2.1. 

2 

Conduct Area 

Reconnaissanc 

e 

Ml 

0.2 

Hrs 

From receipt of tasking, unit 
reconnaissance assets in place. 

2.2.1. 

2 

Conduct Area 

Reconnaissanc 

e 

M2 

Y 

Y/N 

Provide photographic and 
descriptive data of the Named Area 
of Interest to the Commander and 

staff. 

2.2.1. 

3 

Conduct Zone 

Reconnaissanc 

e 

Ml 

0.5 

Hrs 

From receipt of tasking, unit 
reconnaissance assets in place. 
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2.2.1. 

3 

Conduct Zone 

Reconnaissanc 

e 

M2 

N 

Y/N 

Provide photographic and 
descriptive data of the Named Area 
of Interest (NAI) to the Commander 
and staff. 

2.2.5. 

2 

Conduct 

Aviation 

Reconnaissanc 

e 

M3 

34 

% 

Of equipment ready and available 
to provide air reconnaissance 
operations. 

2.2.5. 

2 

Conduct 

Aviation 

Reconnaissanc 

e 

M4 

Y 

Y/N 

Product (sensor) 

dissemination/distribution network 
available. 

2.2.5. 

2 

Conduct 

Aviation 

Reconnaissanc 

e 

M7 

N 

Y/N 

Able to communicate relevant 
reconnaissance information using 
line-of-site (LOS)/beyond-line-of- 
site (BLOS) means. 

2.7 

Conduct 

Ground 

Reconnaissanc 

e and 

Surveillance 

M2 

45 

% 

Of equipment ready and available 
to provide reconnaissance and 
surveillance operations (i.e., 
communications, target 
designation, crew served weapons, 
infiltration/exfiltration equipment, 
mobility assets). 

2.7 

Conduct 

Ground 

Reconnaissanc 

e and 

Surveillance 

M4 

1 

Hrs 

From receipt of tasking, unit 
reconnaissance/surveillance assets 
in place. 

2.7 

Conduct 

Ground 

Reconnaissanc 

e and 

Surveillance 

M5 

70 

% 

Of collection requirements fulfilled 
by reconnaissance/surveillance 
assets. 

2.2 

Collect Data 

and 

Intelligence 

Ml 

25 

% 

Of targets accurately identified. 

2.2 

Collect Data 

and 

Intelligence 

M2 

25 

% 

Of targets accurately located. 
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APPENDIX D. GANTT CHART OUTLINE 


Task Name 

▼ 

Duration 

▼ 

Start 

▼ 

Finish 

▼ 

LTA 2 

5 days 

Mon 4/18/16 

Fri 4/22/16 

” LTA 3 Preparation 

255 days 

Mon 4/25/16 

Fri 4/14/17 

OutdoorTransition/Software Development 

205 days 

Mon 4/25/16 

Fri 2/3/17 

Acceptance Testing 

5 days 

Mon 2/6/17 

Fri 2/10/17 

Correction Time 

15 days 

Mon 2/13/17 

Fri 3/3/17 

Follow- on Acceptance Testing 

5 days 

Mon 3/6/17 

Fri 3/10/17 

Correction Time 

25 days 

Mon 3/13/17 

Fri 4/14/17 

LTA 3 

5 days 

Mon 4/17/17 

Fri 4/21/17 

~ LTA 4 Preparation 

130 days 

Mon 4/24/17 

Fri 10/20/17 

Coactive Design 

30 days 

Mon 4/24/17 

Fri 6/2/17 

Model Driven Software Development (MDSD) 

30 days 

Mon 4/24/17 

Fri 6/2/17 

Voice Command Recognition 

50 days 

Mon 6/5/17 

Fri 8/11/17 

Acceptance Testing 

5 days 

Mon 8/14/17 

Fri 8/18/17 

Correction Time 

25 days 

Mon 8/21/17 

Fri 9/22/17 

Follow- on Acceptance Testing 

5 days 

Mon 9/25/17 

Fri 9/29/17 

Correction Time 

15 days 

Mon 10/2/17 

Fri 10/20/17 

LTA 4 

5 days 

Mon 10/23/17 

Fri 10/27/17 

” LTA 5 Preparation 

125 days 

Mon 10/30/17 

Fri 4/20/18 

Coactive Design/MDSD 

30 days 

Mon 10/30/17 

Fri 12/8/17 

Auto-Follow Development 

45 days 

Mon 12/11/17 

Fri 2/9/18 

Hand and Arm Signal Recognition 

45 days 

Mon 12/11/17 

Fri 2/9/18 

Acceptance Testing 

5 days 

Mon 2/12/18 

Fri 2/16/18 

Correction Time 

21 days 

Fri 2/16/18 

Fri 3/16/18 

Follow- on Acceptance Testing 

5 days 

Mon 3/19/18 

Fri 3/23/18 

Correction Time 

20 days 

Mon 3/26/18 

Fri 4/20/18 

LTA 5 

5 days 

Mon 4/23/18 

Fri 4/27/18 

' LTA 6 Preparation 

125 days 

Mon 4/30/18 

Fri 10/19/18 

Coactive Design/MDSD 

30 days 

Mon 4/30/18 

Fri 6/8/18 

Threat Analysis 

45 days 

Mon 6/11/18 

Fri 8/10/18 

Self Diagnostic 

45 days 

Mon 6/11/18 

Fri 8/10/18 

Acceptance Testing 

5 days 

Mon 8/13/18 

Fri 8/17/18 

Correction Time 

20 days 

Mon 8/20/18 

Fri 9/14/18 

Follow- on Acceptance Testing 

5 days 

Mon 9/17/18 

Fri 9/21/18 

Correction Time 

20 days 

Mon 9/24/18 

Fri 10/19/18 

LTA 6 

5 days 

Mon 10/22/18 

Fri 10/26/18 
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APPENDIX E. LTA WORKSHEET 


UTACC LTA 3 

May 2016 

References 

(1) COBP Campaigns of Experimentation 

(2) COBP Experimentation 

(3) LTA 2 Scenario Tasks 

Enclosures 

(1) MCT List 

Task org: 

Marine Corps Warfighting Lab 


SITUATION: Marine Corps Warfighting Lab (MCWL) has tasked Naval 

Postgraduate School (NPS) and industry partners with the development 
of the Unmanned Tactical Autonomous Control and Collaboration system. 
To accomplish this task a series of Limited Technical Assessments 
(LTA) and Limited Objective Experiments (LOE) are needed to create a 
viable system that meets the requirements of MCWL and enhance the 
warfighting ability of the Marine Corps as a whole. These LTAs and 
LOEs will follow the fundamental concepts of variety and replication 
as put forward in the Code of Best Practices Campaigns of 
Experimentation. 


MISSION: NLT 17 April 2017, 

MCWL sponsors LTA 3, location 
to be determined in order to 
replicate LTA 2 performance 
accomplishments, and advance 
the UTACC system in an outdoor 
environment. 


Intent: 

1. Purpose. The purpose of LTA 3 is 
to take the UTACC system from an indoor 
controlled environment to an outdoor 
controlled environment while further 
testing the capabilities of the system 
interface, onboard sensors, and 
software as well as new robotic 
platforms. 


2. Method. Having met the 
requirements for a) UTACC software 
utilization in the GUSS autonomous 
system and Phoenix UAV, and b) 
demonstration of successful outdoor 
transition in acceptance and follow-on 
testing, LTA 3 will be conducted in an 
outdoor environment with a simulated 
urban setting. This venue will allow 
the system developers to replicate 


57 




COI: 

1) Will the system reduce 
the cognitive load of 
the team? 

2 ) Will the system render 
enhanced 3-Dimensional 
reconnaissance products 

3) Will the system increas 
the safety of the team? 

4) Will the system enhance 
identification and 
engagement of targets? 

5) Does the system operate 
in accordance with 
Marine Corps doctrine? 

6 ) To what extent does the 
digital plan provide 
context to the machines 
as well as the Marines? 

7) Does the system 
demonstrate flexibility 
to changes in the 
environment/plan? 


EXECUTION: 

Concept of Operations: 

1. On 17 April 2017, required personnel involved in the UTACC 
system development will arrive at a testing and evaluation 
location selected by MCWL. The desired location for testing and 
evaluation is a relatively flat, outdoor training area with a 
simulated urban setting. This environment will allow the system 
to be newly introduced to an outdoor environment that will not 
provide unnecessary mobility challenges at such an early stage 
of development. This environment will simultaneously provide a 
setting that facilitates the advancement and replication of 
previous LTA accomplishments. The environment will also 
facilitate testing of the capabilities of the GUSS autonomous 
vehicle and Phoenix 90 UAV under the control of UTACC software 
and interfaces. 

2. Testing will be broken into three phases. Phase I will begin 
with a Marine operator inputting mission parameters and 
releasing the system to begin reconnaissance. The system will 
conduct a reconnaissance of an identified area of interest 
(AOI), building a 3D map as it does so. Upon recognizing that 
it cannot complete the mission as required, the system alerts 
the Marine and the Marine authorizes the launch of the systems 
UAV to assist in the completion of the mission. When 
reconnaissance is completed, the system requests permission from 
the Marine to return to base (RTB). When the Marine operator 
gives authorization, the system will return to its original 
start point. Exit criteria for Phase I will be that all 
performance standards established in MCTs 1, 1.5, and 2 are met. 


results from LTA 2- specifically MCTs 
1, 1.5, and 2- while advancing the 
capabilities demonstrated in these MCTs 
in an outdoor environment. LTA 3 will 
be conducted in three phases utilizing 
the GUSS autonomous vehicle and Phoenix 
90 UAV as robotic platforms. The 
system will be tasked with creating a 
3D model of the environment, facial 
recognition of a person of interest, 
reaction to a newly introduced variable 
in the environment, and 
deriving/building a digital plan for 
the approach to the target. 

3. End state. The UTACC system 
demonstrates the ability to meet 
previously tested MCT performance 
requirements in an outdoor environment 
while identifying shortfalls/ advancing 
the capabilities of the system 
interface, onboard sensors, and 
software. 
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3. Phase II will replicate the results of Phase I with the addition 
of the system being required to conduct facial recognition of a 
person and/or object of interest. After receiving verification 
from the Marine operator that the system has accurately 
recognized the target, the system will request permission to 
engage. When the Marine operator authorizes engagement, the 
system will relay a request for fire to a "firing element." 

Exit criteria for Phase II are that previous MCTs are met and 
successful target recognition and engagement has been completed. 

4. Phase III will begin at the completion of Phase II and will 
accomplish the performance goals of Phases I and II while 
reacting effectively to the introduction of a new variable and 
demonstrating basic planning capabilities. Utilizing the 
currently generated 3D map of the AOI, the system will be 
required a deliver a plan for approval, disapproval, or 
modification to the Marine operator. The plan will orient the 
Marine to the AOI and suggest a potential approach route to an 
operator selected waypoint. Also in this phase, a vehicle, 
previously uploaded to the system as a BOLO, will enter the AOI. 
The system must accurately identify the vehicle as the BOLO 
vehicle, notify the Marine operator for verification, and 
request guidance for follow-on action. Follow-on action can 
consist of targeting, observation utilizing either the UGV or 
UAV (the decision for asset usage will be left to the Marine 
operator), or to ignore the vehicle and continue with previous 
mission tasking. 


5. Phase III exit criteria are as follows; 


a. Successful completion of MCTs 1, 1.5, and 2, 

b. Successful facial recognition and engagement of a target, 

c. Successful identification of a newly introduced BOLO 
vehicle with requests for action; targeting, observation 
utilizing either the UGV or UAV, or to ignore the vehicle 
and continue with previous mission tasking, 

d. A mission plan, created from the 3D map, utilizing a 
Marine operator selected waypoint is successfully 
generated for approval, disapproval, or modification. 


6. LTA 3 is complete when all three phases and their associated 
entrance and exit criteria are met. 
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